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This  is  a  summary  of  research  completed  during  the  period  of  this  award, 
1  October  1985  -  30  September  1988  by  Fleming,  Kushner  together  with  asso¬ 
ciated  postdoctoral  and  graduate  student  personnel.  The  research  covers  a 
number  of  problems  in  many  areas  of  stochastic  control,  recursive  stochastic 
algorithms,  and  related  areas  of  analysis.  It  is  part  of  a  continuing  research 
program  pursued  successfully  for  a  number  of  years.  The  program  has  been 
motivated  both  by  traditional  applications  in  control  and  filtering  and  by 
newer  areas  of  application  arising  in  queueing/communication  networks  and 
production  systems.  Other  research  issues  addressed  include  numerical  meth¬ 
ods  for  stochastic  control  and  recursive  algorithms  for  distributed  and  parallel 
processing  and/or  control. 

The  work  of  Fleming  and  Kushner  will  be  summarized  in  turn,  with 
references  to  research  publications  supported  under  this  award. 


W.  H.  Fleming 


Fleming’s  research  was  concerned  with  optimal  stochastic  control  the¬ 
ory,  nonlinear  filtering,  large  deviations  for  Markov  diffusions,  and  viscosity 
solutions  of  nonlinear  partial  differential  equations. 

Fleming’s  work  in  optimal  stochastic  control  focused  on  control  of  Markov 
diffusion  processes,  with  complete  or  partial  observations.  This  work  is  re¬ 
ported  in  references  [F3]  [F4]  [Fo]  [Fll  -  F12],  [F3]  [12]  provide  a  solution  to 
a  stochastic  production  planning  problem,  making  use  of  viscosity  solution 
methods  for  a  system  of  first-order  nonlinear  partial  differential  equations. 

[F4]  [Fll]  provide  an  approximate  solution,  in  the  form  of  an  asymptotic  se¬ 
ries  in  powers  of  a  small  parameter  measuring  the  intensity  of  noise  entering 
the  control  dynamics.  References  [F5]  [F13]  provide  an  alternative  to  the 
usual  dynamic  programming,  seeking  maximal  smooth  subsolutions  rather 
than  solutions  to  the  dynamic  programming  equations  and  using  an  abstract 
duality  theorem  from  convex  analysis. 

The  theory  of  large  deviations  is  concerned  with  asymptotic  estimates 

for  exponentially  small  probabilities  of  rare  events  associated  with  stochas-  _ _ _ 

tic  processes.  In  1977,  Fleming  introduced  a  stochastic  control  approach  to  ,lon  *or 
large  deviations  of  nearly  deterministic  Markov  diffusions.  This  technique  GRA4I 
was  based  on  a  certain  logarithmic  transformation.  Subsequently,  simpler 
analytical  methods  based  on  logarithmic  transformations  and  viscosity  solu-  icatloa. 
tion  techniques  were  developed  [ F 1  ] .  These  methods  also  give  more  accurate  - - 
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approximations  in  the  form  of  an  asymptotic  series  [F4][F1 1] .  These  analyt¬ 
ical  methods  were  adapted  to  other  classes  of  Markov  processes  in  [F8]. 

Fleming  and  Souganidis  [F7][F14]  developed  a  theory  of  value  for  two- 
player,  zero  sum  stochastic  differential  games.  This  provides  an  optimization 
formula  for  viscosity  solutions  to  a  large  class  of  nonlinear  second  partial 
differential  equations,  of  degenerate  parabolic  type. 

References  [F9]  [F10]  [F15]  are  concerned  with  nonlinear  filtering,  in  cases 
when  a  namy-to-one  function  function  of  the  system  state  plue  low  intensity 
observation  noise  is  observed.  The  goal  is  to  obtain  finite-dimensional,  ap¬ 
proximately  optimal  filters. 

Research  Publications  W.H.  Fleming 
(Fll  ■  [F15] 

FI.  A  PDE  Approach  to  Asymptotic  Estimates  for  Optimal  Exit  Proba¬ 
bilities  (with  P.E.  Souganidis),  Annali  della  Scuola  Normale  Superiore 
Pisa,  Ser.  IV  21  (1986)  171-192. 

F  2.  Stochastic  Variational  Formula  for  Fundamental  Solutions  of  Parabolic 
PDE  (  with  S-J  Sheu),  Applied  Math  and  Optimization,  El  (1985). 

F  3.  An  Optimal  Stochastic  Production  Planning  Problem  with  Randomly 
Fluctuating  Demand,  (with  S.P.  Sethi  and  H.M.  Soner),  SIAM  J.  on 
Control  and  Optimization,  21  (1987).  1494-1502. 

F  4.  Asymptotic  Series  and  the  Method  of  Vanishing  Viscosity  (  with  P.E. 
Souganidis),  Indiana  Univ.  Math.  J.  2&  (1986)  425-447. 

F  5.  Convex  Duality  Approach  to  the  Optimal  Control  of  Diffusions  (with 
D.  Vermes),  SIAM  J  on  Control  and  Optimiz  to  appear  in  1989. 

F  6.  On  the  Existence  of  the  Dominant  Eigenvalue  and  its  Application  to 
the  Large  Deviation  Properties  of  an  Ergodic  Markov  Process  (  with 
S.  J.  Sheu  and  H.M.  Soner),  22  (1987)  187-199. 

F  7.  Value  Functions  for  Two-Plaver,  Zero-Sum  Stochastic  Differential  Games 
(with  P.E.  Souganidis),  Indiana  Univ.  Math  J,  to  appear  in  19S9. 


2 


F  8.  Asymptotic  Expansions  for  Markov  Processes  with  Levy  Generators 
(with  H.  M.  Soner),  submitted  to  Applied  Math,  and  Optimiz,  19 
(1989)  203-223. 

F  9.  Piecewise  monotone  filtering  with  small  observation  noise  (with  E.  Par- 
doux),  SIAM  J.  on  Control  and  Optimiz  (1989). 

F  10.  Discrete  time  piecewise  linear  filtering  with  small  observation  noise 
(with  D.  Ji,  P.  Salame,  Q.  Zhang)  LCDS/CCS  Report  No.  88-27. 

F  11.  Asymptotic  Series  for  Solutions  to  the  Dynamic  Programming  Equa¬ 
tion  for  Diffusions  with  Small  Noise,  (with  P.E.  Souganidis)  Proc.  24th 
IEEE  Conf.  on  Decision  and  Control,  Ft.  Lauderdale,  Florida,  Decem¬ 
ber  11-13,  Vol.  1,  1985. 

F  12.  A  Stochastic  Production  Planning  Problem  with  Random  Demand, 
(with  H.M.  Soner)  Proc.  24th  IEEE  Conf.  on  Decision  and  Control, 
Vol.  1,  Ft.  Lauderdale,  Florida,  December  11-13,  1985. 

F  13.  Generalized  Solutions  in  the  Optimal  Control  of  Diffusions  (with  D. 
Vermes)  Proc.  IMA  Workshop,  June  1986,  IMA  Vols.  in  Math,  and 
Applic.  No.  10,  Springer- Verlag,  1987,  119-127. 

F  14.  Two-Player,  Zero-Sum  Stochastic  Differential  Games  (with  P.E.  Sougani¬ 
dis)  Proc.  of  Conf.  in  honor  of  J.L.  Lions,  June  1988. 

F  15.  Piecewise-linear  filtering  with  small  observation  noise  (  with  D.  Ji  and 
E.  Pardoux)  Proc.  8tii  INRLA  Conf.  on  Analysis  and  Optimiz.  of 
Systems,  Springer  Lect.  Notes  in  Control  and  Info.  Sci:  No.  Ill,  1988. 

H.J.  Kushner 

Kushner’s  research  covered  a  wide  range  of  topics  in  stochastic  systems 
theory  and  applied  probability.  These  including:  large  deviation  with  com¬ 
munications  applications,  stochastic  approximations  (  convergence  theorems, 
large  deviations  estimates),  adaptive  filters,  distributed  parameter  stochastic 
systems,  wide  band  noise  approximations,  Monte  Carlo  methods,  distributed 
and  communicating  stochastic  approximation  algorithms,  singular  stochastic 
control  and  computational  methods  in  optimal  stochastic  control. 
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The  work  on  large  deviations  and  applications  is  reported  in  [K4]  [KS] 
[K16]  [K17].  Typical  communications  applications  arise  in  models  with  rapidly 
varying  noise  inputs,  for  slowly  adapting  digital  systems,  and  for  tracking 
systems  with  small  noise  effects.  Among  the  accomplishments  is  a  “quick 
simulation”technique,  based  on  a  change  of  probability  measure  technique. 
This  method  relies  on  the  numerical  solution  of  a  first-order  nonlinear  partial 
differential  equation,  connected  with  the  action  functional  being  minimized 
to  obtain  the  large  deviation  rate. 

The  papers  on  stochastic  approximation  and  recursive  algorithms  [K5] 
[KlO]  [Kll]  [K21]  provide  limit  theorems  and  large  deviations  estimates  under 
conditions  on  dynamics  and  noise  which  are  broad  enough  to  fit  most  current 
applications  in  control  and  communication  theory.  The  global  behavior  of 
stochastic  approximations  was  studied  by  Monte  Carlo  methods  in  [K7]. 

Ivushner’s  work  on  distributed  parameter  stochastic  systems,  described  by 
stochastic  partial  differential  equations,  is  reported  in  [K2]  [K3].  This  work  is 
concerned  with  such  questions  as  stability  and  near-stationarity  for  systems 
with  wide-band  noise  inputs,  and  with  nonlinear  filtering  applications. 

Further  work  on  wide-band  noise  approximations  in  filtering  and  stochas¬ 
tic  control  is  reported  in  [I<6]  [K9]  [K14]  [K15]  [K24].  In  the  part  of  this  work 
concerned  with  stochastic  control,  the  problem  is  to  find  nearly-optimal  con¬ 
trol  laws  for  a  wide-band  noise  driven  system,  based  on  optimal  control  laws 
for  an  idealized  white-noise  driven  system  (  for  which  there  is  an  extensive 
theory.) 

Recent  work  by  Kushner  on  stochastic  approximation  [I\12]  [K13]  con¬ 
cerns  distributed  and  communicating  systems,  in  which  processors  are  located 
at  physically  distinct  sites. 

Routing  and  flow  control  problems  for  queues  under  heavy  traffic  can 
be  analyzed  using  methods  of  singular  stochastic  control.  Such  problems, 
and  related  problems  for  wide-band  noise  driven  singular  stochastic  control 
systems  were  analyzed  in  [K1S]  [K19][20] 

Ref.  [22]  is  an  invited  survey  of  recent  developments  on  numerical  meth¬ 
ods  for  stochastic  control  problems.  It  complements  Kushner’s  earlier  book 
which  dealt  with  this  topic  “Probability  Methods  for  Approximation  in  Stochas¬ 
tic  Control  and  for  Elliptic  Equations  ”,  1977. 
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Research  Publications  -  H.J.  Kushner 


K  1.  Approximating  multiple  Ito  integrals  with  “band-limited”  processes, 
Stochastics,  M,  1985,  pp  85-114. 

I<  2.  Weak  convergence  approximations  for  partial  differential  equations  with 
stochastic  coefficients,  Stochastics,  14.,  1985,  pp  115-148. 

K  3.  Asymptotic  properties,  stability  and  “near”  stationanity  of  parabolic 
partial  differential  equations  with  wide  bandwidth  inputs,  Stochastics, 
1£,  1986,  pp  111-136. 

I<  4.  Large  deviations  estimates  for  systems  with  small  noise  effects,  and 
applications  to  stochastic  systems  theory,  (with  P.  DuPuis)  SIAM  J. 
on  Control  and  Optimization,  21,  1986,  pp  979-1008. 

K  5.  The  theory  of  large  deviations  and  asymptotic  analysis  of  recursive  al¬ 
gorithms  and  stochastic  approximation  (with  P.  DuPuis),  in  Advances 
in  Statistical  Signal  Processing,  ed.  by  H.V.  Poor,  JAI  Press. 

I<  6.  Approximate  and  limit  results  for  nonlinear  filters  with  wide  bandwidth 
observation  noise,  Stochastics,  ]£,  1986,  pp  65-96. 

I\  7.  Asymptotic  global  behavior  for  stochastic  approximations  with  slowly 
decreasing  gain:  Global  optimization  via  Monte-Carlo,  in  SIAM  J.  on 
Appl.  Math.,  U,  1987,  169-185. 

K  8.  Stochastic  systems  with  small  noise,  analysis  and  simulation,  a  phase 
locked  loop  example,  SIAM  J.  Appl.  Math.,  U,  1987,  643-661. 

K  9.  Nearly  optimal  state  feedback  controls  for  stochastic  systems  with  wide¬ 
band  noise  disturbances,  (with  W.  Runggaldier)  SIAM  J.  Control  and 
Optimization,  2H,  19S7,  298-315. 

K  10.  Constrained  stochastic  approximation  by  the  theory  of  large  deviations, 
‘Robbins  Symposium  Volume’,  published  by  Wiley. 

K  11.  Asymptotic  behavior  of  constrained  stochastic  approximations  via  the 
theory  of  large  deviations.  Z.  Wahrscheinlichkeitsteorie,  7£.19S7,  p  224- 
244. 
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K  12.  Asymptotic  properties  of  distributed  and  communicating  stochastic  ap¬ 
proximation  algorithms  (with  G.  Yin),  in  SIAM  J.  Control  and  Opti¬ 
mization,  25.,  1987;  1266-1290. 

I\  13.  Stochastic  approximation  algorithms  for  parallel  and  distributed  pro¬ 
cessing  (  with  G.  Yin),  Stochastics,  22.,  1987,  p  219-250. 

K  14.  Filtering  and  control  for  wide  bandwith  noise  driven  systems  IEEE 
Trans,  on  Automatic  control,  T-AC87,  1987,  p  123-133. 

K  15.  Almost  optimal  controls  for  wide  band  noise  driven  systems,  Institute 
of  Math,  and  Appl.  (Minnesota)  Volume  10,  pub.  by  Springer,  1987  ( 
W.  Fleming,  P.L.  Lions,  ed.). 

I<  16.  Minimizing  escape  probabilities:  a  large  deviations  approach  (with  P. 
DuPuis)  to  appear  SIAM  J.  on  Control  and  Optimization. 

I<  17.  Upper  bounds  for  large  deviations  for  non-smooth  stochastic  difference 
eqns.,  LCDS  Dept.  87-8  (Feb.  87)  sub.  to  App.  Prob. 

I<  18.  Nearly  optimal  singular  controls  for  wideband  noise  driven  systems, 
(with  K.  M.  Ramachandran),  SIAM  J.  on  Control  and  Optimization 
[26],  p  569-591,  1988. 

I\  19.  Optimal  and  approximately  optimal  control  policies  for  queues  in  heavy 
traffic  (  with  K.M.  Ramachandran)  to  appear  SIAM  J.  on  Control  and 
Optimization. 

I\  20.  Diffusion  approximations  and  nearly  optimal  maintenance  policies  for 
system  breakdown  repair  policies,  to  appear  in  Appl.  Math,  and 
Optimization. 

I\  21.  Stochastic  approximation  and  large  deviations:  General  results  for  w. 
p.  1  convergences,  (with  P.  DuPuis)  to  appear  SIAM  J.  on  Control  and 
Optimization,  LCDS  Dept.  87-21,  19S7. 

K  22.  Numerical  methods  for  stochastic  control  problems  in  continuous  time, 
invited  paper  SIAM  .1.  un  Control  and  Optimization. 
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